Identifying facial expressions in acquired digital images

ABSTRACT

A face is detected and identified within an acquired digital image. One or more features of the face is/are extracted from the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both. A model including multiple shape parameters is applied to the two independent eyes or subsets of features of each of the two eyes, and/or to the lips or partial lips or one or more other mouth features and one or both eyes. One or more similarities between the one or more features of the face and a library of reference feature sets is/are determined. A probable facial expression is identified based on the determining of the one or more similarities.

PRIORITY AND RELATED APPLICATIONS

This application claims the benefit of priority to U.S. provisionalpatent applications Nos. 61/179,808, filed May 20, 2009 and 61/221,417,filed Jun. 29, 2009 and 61/221,425, filed Jun. 29, 2009. Thisapplication is related to U.S. patent application Ser. Nos. 12/038,147,published as 20080205712, and 12/203,807, published as 20090003661, and11/027,001, now U.S. Pat. Nos. 7,715,597, and 11/833,224, now U.S. Pat.Nos. 7,515,740, and 11/761,647, published as 20080013798. Each of thesepatents and patent applications is incorporated by reference.

BACKGROUND

In the recent past, a digital camera was only able to capture images.Recently, however, face-tracking technology has become a common featureof consumer digital cameras, e.g., U.S. Pat. Nos. 7,403,643, 7,460,695,7,315,631 and U.S. application Ser. Nos. 12/063,089 and 12/479,593 areincorporated by reference. The most recent implementations featurehardware coded face tracking in an IP-core. Applications to date havebeen limited to optimizing the exposure and acquisition parameters of afinal image. Yet there are many additional applications which have evengreater potential to enrich the user experience. The rapid deployment offace tracking technology in cameras suggests that other more advancedface analysis techniques will soon become feasible and begin to offereven more sophisticated capabilities in such consumer devices.

The detailed analysis of facial expression is one such technique whichcan offer a wide range of new consumer applications for mobile andembedded devices. In the context of managing our personal imagecollections, it is useful to be able to sort images according to thepeople in those images. It would be even more useful if it were possibleto determine their emotions and thus enable images to be further sortedand categorized according to the emotions of the subjects in an image.

Other consumer device applications also can gain from such capabilities.Many such devices now feature a camera facing the user, e.g. most mobilesmartphones, and thus user-interfaces could respond directly to ourfacial expressions as set forth in certain embodiment of the presentinvention. In one example, an e-learning system could match its level ofdifficulty to the degree of puzzlement on the student's face, forexample. In another example, a home health system could monitor thelevel of pain from an elderly person's facial expression. In furtherexamples, other domains such as entertainment and computer gaming,automotive, or security can also benefit from such applications. As theunderlying expression recognition technologies improve in accuracy, therange of applications will grow further.

Computer gaming has grown from its humble origins to become a globalindustry rivaling the movie industry in terms of scale and economicimpact. The technology of gaming continues to improve and evolve at avery rapid pace both in terms of control interface and the graphicaldisplay of the gaming world. Today's user interfaces feature moresophisticated techniques for players to interact and play co-operativelywith one another. It is possible to have real-time video and audio linksbetween the real players so they can co-ordinate their group gameplay.

However the emphasis remains on the player being drawn into theartificial game world of the computer. There is still little scope forthe conventional gaming environment to reach back to the players,sensing and empathizing with their moods and feelings. Given thesophistication of modern AI game engines, it is believed that this is amissed opportunity and that gaming engines can be advantageously evolvein accordance with embodiments described below to develop and providemethods to empathize with individual game players.

Y. Fu et al have presented a framework of multimodal human-machine orhuman-human interaction via real-time humanoid avatar communication forreal-world mobile applications (see, e.g., Hao Tang; Yun Fu; Jilin Tu;Hasegawa-Johnson, M.; Huang, T. S., “Humanoid Audio—Visual Avatar WithEmotive Text-to-Speech Synthesis,” Multimedia, IEEE Transactions on,vol. 10, no. 6, pp. 969-981, October 2008, incorporated by reference).Their application is based on a face detector and a face tracker. Theface of the user is detected and the movement of the head is trackeddetecting the different angles, sending these movements to the 3Davatar. This avatar is used for low-bit rate virtual communication. Adrawback of this approach is that the shape of the avatar needs to bespecified by the user and forward-backward movement of the user is notdetected, so the avatar appears as a fixed-distance portrait in thedisplay.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 schematically illustrates a facial analysis and classificationsystem in accordance with certain embodiments.

FIG. 2 illustrates several examples of detected faces using theViola-Jones algorithm.

FIG. 3 illustrates an example of poor fitting for winkingin-plane-rotated eyes in accordance with certain embodiments.

FIG. 4 a-4 c illustrate an annotation for the global AAM, and for twosub-models: for open and closed eyes, respectively.

FIG. 5 illustrates a fitting algorithm for open and closed eyessub-models in accordance with certain embodiments.

FIG. 6 a-6 c illustrate annotation for a global model, for a localsub-model, and its mirroring in order to obtain the right eye,respectively, in accordance with certain embodiments.

FIGS. 7 a-7 c illustrate fitting the standard AAM model, fitting theleft/right eye sub-models without refitting the global model, andfitting the left/right eye sub-models and refitting the global model,respectively, in accordance with certain embodiments.

FIG. 8 illustrates a fitting algorithm for a component based AAM eyemodel in accordance with certain embodiments.

FIGS. 9 a-9 b illustrate a comparison of two proposed component-basedversions: the two-eyes sub-model vs. the single-eye sub-model,respectively, in accordance with certain embodiments.

FIG. 10 illustrates a histogram of the boundary error for the threealgorithms: conventional AAM, sub-model fitting and the component-basedAAM.

FIGS. 11 a-11 c illustrate lip region pre-processing and show anoriginal image, an after the hue filter representation, and an after thebinarization representation, in accordance with certain embodiments.

FIG. 12 illustrates a lip modeling system overview in accordance withcertain embodiments.

FIG. 13 illustrates full component-AAM Face Model incorporating bothimproved Eyes and Lips models in a single component-AAM framework.

FIGS. 14 a-14 c illustrate examples of shape fitting on an unseenpicture with expression and pose variations in accordance with certainembodiments.

FIG. 15 shows several plots that illustrate means over shape parametersfor each of the seven universal emotions.

FIG. 16 shows a plot of FER accuracy versus the number of shapeparameters.

FIG. 17 illustrates different FER classification schemes comparingrecognition rates with and without the lowest order AAM shape parameter.

FIG. 18 illustrates on each row examples of the six universal facialexpressions and the neutral state, as expressed by different subjects(in order, anger, happiness, neutral, surprise, fear, sadness, anddisgust).

FIGS. 19 a-19 b illustrate performances of expression recognition of SVMclassifiers in a cascade structure for MMI and FEEDTUM databases, andillustrate cascaded structures for each database including the mosteffective six classifiers of the seven which classify the six universalexpressions, and the neutral expression, wherein the recognition rate isprovided after each stage of the cascade.

DETAILED DESCRIPTIONS OF THE EMBODIMENTS

Techniques of recognizing a facial expression or an identity of a face,or both, are provided. A face is detected and identified within anacquired digital image. The technique involves separately extracting oneor more features of the face within the digital image, including twoindependent eyes or subsets of features of each of the two eyes, or lipsor partial lips or one or more other mouth features and one or botheyes, or both. A model is applied including multiple shape parameters tothe two independent eyes or subsets of features of each of the two eyes,and/or to the lips or partial lips or one or more other mouth featuresand one or both eyes. One or more similarities are determined betweenthe one or more features of the face and a library of reference featuresets. A probable facial expression is identified based on thedetermining of the one or more similarities.

The extracting separately of the one or more features of the face may beperformed before the identifying of the face.

The one or more features may include one or more geometric features. Theone or more geometric features may include one or more shapes,deformations or locations of facial components, and/or pose variations.

The one or more features may include one or more appearance features.The one or more appearance features may include one or more skin texturechanges-such as one or more furrows, bulges, expression wrinkles, orillumination variations, and/or blushing.

The library may include face features of faces captured in multiple posevariations and/or under multiple directional illumination conditions.

The facial expression may be categorized as indicating surprise, fear,happiness, anger, neutral, sadness, disgust, or a combination thereof.

A game difficulty and/or workflow may be adapted based on the identifiedprobable facial expression. The identified probable facial expressionmay be mirrored in an avatar within a game display.

The model may include an Active Appearance Model (AAM).

The multiple shape parameters may include seven or fewer shapeparameters, such as six or seven.

A digital image acquisition device may be configured to recognize afacial expression or an identity of a face, or both, including any ofthe features described herein. The device may include a lens and imagesensor for capturing a digital image, and a processor. A face detectionand recognition module may be configured to program the processor todetect and identify a face within the digital image. A featureextraction module may be configured to program the processor to extractone or more features of the face within the digital image, including twoindependent eyes or subsets of features of each of the two eyes, or lipsor partial lips or one or more other mouth features and one or botheyes, or both. A classification module may be configured to program theprocessor to determine one or more similarities between the one or morefeatures of the face and a library of reference feature sets. A facialexpression identification module may be configured to program theprocessor to identify a probable facial expression based on thedetermining of the one or more similarities.

One or more computer-readable media are also provided that have codeembedded therein for programming a processor to perform a method ofrecognizing a facial expression or an identity of a face, or both, inaccordance with any of the described techniques.

An advantageous modelling approach is provided for improveddetermination of facial expressions captured for example from alow-resolution video stream. An extension of an Active Appearance Model(AAM) is provided in certain embodiments to measure facial parameters,and a set of classifiers to determine facial states are described.

Embodiments for this technique are described for example to determinethe emotional state of the players of a computer game and suggest howthis information can be integrated into the workflow of a game. A numberof other uses are described where the game environment can be adaptedbased on feedback from the players.

A direct connection back to the game player is also provided in certainembodiments. An advantageous approach to modelling and classifying thefacial expressions of a game player is provided that uses alow-resolution webcam and state-of-art face detection and face modellingtechniques. The system is capable of running in real-time on a standarddesktop PC or other processor-enabled consumer appliance, such as on adedicated embedded device, or converted to a dedicated hardwaresubsystem.

System Architecture

A system that performs automatic face recognition or expressionrecognition may typically comprise three main subsystems, as shown inFIG. 1: (i) a face detection module, (ii) a feature extraction module,and (iii) a classification module which determines a similarity betweenthe set of extracted features and a library of reference feature sets.Other filters or data pre-processing modules can be used between thesemain modules to improve the detection, feature extraction orclassification results.

FIG. 1 schematically illustrates a facial analysis and classificationsystem in accordance with certain embodiments. The system according tothe example of FIG. 1 includes a still and/or video image acquisitioncomponent 2. A raw data block 4 extracts raw image data from imagesacquired by the image acquisition component 2. The raw data is receivedat a face detection module 6. The face detection module 6 can includeface tracking and/or pose estimation 8 and/or the face detection module6 may work together with a separate face tracking and/or pose estimationmodule 8. A feature extraction module 10 may then include featuretracking and/or gaze tracking 12 or there may be a separate featuretracking and/or gaze tracking module 12. The system of the example ofFIG. 1 then includes a face recognition module 14 and an expressionrecognition module 16. The face recognition module 14 may performidentification or verification as to whom a detected face specificallybelongs to. The expression recognition module may perform classificationor recognition of a specific facial expression or facial featureexpression. These face recognition and expression recognition modules 14and 16, respectively, may include or work in combination with consumerapplications 18.

In the face detection module 6, it is decided whether the input pictureor the input video sequence contains one or more faces. If so, thenfacial features are extracted from the detected faces using the featureextraction module 10, for example, by applying an advanced face modelwhich encodes the facial features by a set of parameters. As a nextstep, using the face recognition and/or expression recognition modules14, 16, facial features, determined as a set of output parameters fromthe model, are classified in order to perform facial recognition orexpression classification.

Face Detection Module

In a system in accordance with certain embodiments, a face detectormodule 6 is employed as initialization for an AAM search. Face detectioncan be defined as the ability to detect and localize faces within animage or a scene. In the last few years, many different face detectiontechniques have been proposed in literature. A comprehensive survey ofconventional face detection methods is presented at Yang, M.-H., D. J.Kriegman, and N. Ahuja, Detecting Faces in Images: A Survey. IEEETransactions on pattern analysis and machine intelligence, 2002. 24(1):p. 34-59, incorporated by reference. State-of-the-art face detectionmethods provide real-time solutions that report high detection rates.

In this field, significant advances have been due to the work of PaulViola and David Jones who proposed a face detector based on rectangularHaar classifiers and the integral image representation of an inputimage, e.g., at P. A. Viola, M. J. Jones, “Robust real-time facedetection”, International Journal of Computer Vision, vol. 57, no. 2,pp. 137-154, 2004, incorporated by reference. Such face detectionmethods are among the fastest reported in the literature so far. It isable to perform for semi-frontal faces in real-time and is highlyaccurate. In certain embodiments, the Viola-Jones face detector isemployed as initialization for an AAM search. An example of facedetection using a Viola-Jones based method is shown in FIG. 2. Thisalgorithm has been implemented in OpenCV, as reported, e.g., athttp://www.mathworks.com/matlabcentral/fileexchange/19912, and G.Bradski, A. Kaehler, and V. Pisarevski, “Learning-based computer visionwith intel's open source computer vision library,” Intel TechnologyJournal, vol. 9, no. 2, pp. 119-130, May 2005, which is a free computervision library used widely by the computer vision research community.

Feature Extraction Module

In certain embodiments, the AAM approach is extended for facial featureextraction. AAM can be utilized as a powerful tool for imageinterpretation in accordance with certain embodiments, particularly whendealing with generalized facial expressions. It is a global face modelwhereas the key features which determine facial expressions are localfeatures. It is these local features which are responsible for most ofthe relevant facial variation.

Component-based AAM, e.g., see Zhang and F. S. Cohen Component-basedActive Appearance Models for face Modeling, in International Conferenceof Advances in Biometrics, ICB, Hong Kong, China, Jan. 5-7, 2006,incorporated by reference, offers a practical solution that may becombined with other described features within certain embodiments. Itcombines a global face model with a series of sub-models. Thesesub-models may be typically component parts of the object to be modeled.This approach benefits from both the generality of a global AAM modeland the local optimizations provided by its sub-models. The model may beadjusted through training to be robust to small to medium posevariations and to directional illumination changes.

Expression Classification Module

In various embodiments, generally either of two classifiers are used:Nearest Neighbor (NN) and Support Vector Machine (SVM), e.g., asdescribed at Vishnubhotla, S., Support Vector Classification. 2005, andDasarathy, B. V., Nearest Neighbor (NN) Norms: NN Pattern ClassificationTechniques. 1991, respectively, incorporated by reference. TheEuclidean-NN rule or the cosine-NN rule may be used. These classifiersmay use the relevant AAM parameters in accordance with certainembodiments to choose between facial expressions.

The Extraction of Facial Features Using AAM

Statistical models of appearance such as AAM may be used as a deformablemodel, capable of interpreting and synthesizing new images of an objectof interest. The desired shape to be modelled—in our case a facialregion—is annotated by a number of landmark points. A shape vector isgiven by the concatenated coordinates of all landmark points and may beformally written as, s=(x1, x2, . . . , xL, y1, y2, . . . , yL)T, whereL is the number of landmark points.

The shape model may be obtained by applying Principal Component Analysis(PCA) on the set of aligned shapes, e.g., as follows:

$\begin{matrix}{{s = {\overset{\_}{s} + {\varphi_{s}b_{s}}}},{\overset{\_}{s} = {\frac{1}{N_{s}}{\sum\limits_{i = 1}^{N_{s}}s_{i}}}}} & (1)\end{matrix}$

where s is the mean shape vector, and N_(S) is the number of shapeobservations; φ_(s) is the matrix having the eigenvectors as itscolumns; b_(s) defines the set of parameters of the shape model.

The texture, defined as the pixel values across the object of interest,may also be statistically modelled. In one embodiment, face patches arefirst warped into the mean shape based on a triangulation algorithm.Then a texture vector t=(t1, t2, . . . , tp)^(T) is built for eachtraining image by sampling the values across the shape normalizedpatches. The texture model is also derived by means of PCA on thetexture vectors:

$\begin{matrix}{{t = {\overset{\_}{t} + {\varphi_{t}b_{t}}}},{\overset{\_}{t} = {\frac{1}{N}{\sum\limits_{i = 1}^{N_{t}}t_{i}}}}} & (2)\end{matrix}$

where t is the mean texture vector, with N_(t) as the number of textureobservations; φ_(t) is the matrix of cigenvectors, and b_(t) the textureparameters.

The sets of shape and texture parameters

c=are used in certain embodiments to describe the overall appearancevariability of the modelled object, where W_(s) is a vector of weightsused to compensate the differences in units between shape and textureparameters.

After a statistical model of appearance is created, an AAM algorithm maybe employed in certain embodiments to fit the statistical model to a newimage. This determines the best match of the model to the image allowingto find the parameters of the model which generate a synthetic image asclose as possible to the target image.

Relevant AAM Parameters for Illustrating Emotions

AAM may be used in certain embodiments to extract two types of features:geometric features and appearance features. Geometric features describeshapes, deformations and locations of facial components, and posesvariations. Appearance features describe skin texture changes, e.g.,furrows and bulges, blushing, expression wrinkles, illuminationvariations.

Geometric features are more affected when expressing emotions. Asexamples, when surprised, eyes and mouth open widely, the latterresulting in an elongated chin. When sad, people often blink. Whenangry, eyebrows tend to be drawn together. Both types of features may beadvantageously used to fit an AAM model to unseen pictures. The rightchoice of parameters may serve to provide an optimal determination ofexpression, as described in more detail below.

Relevant Facial Features to Indicate Expressivity

Facial expressions are defined by the dynamics of the individual facialfeatures. Psychophysical experiments (see, e.g., M. Nusseck, D. W.Cunningham, C. Wallraven, H. H. Bülthoff. The contribution of differentfacial regions to the recognition of conversational expressions, Journalof Vision, 8(8):1, pp. 1-23, 2008, incorporated by reference), indicatethat the eyes and the mouth can be generally in most images the mostrelevant facial features in terms of facial expressions. Experimentsshow that, in some cases, one individual facial region can entirelyrepresent an expression. In other cases, it is better to use theinteraction of more than one facial area to clarify the expression.

A thorough description of the eye area is provided using an AAM model atI. Bacivarov, M. Ionita, P. Corcoran, Statistical Models of Appearancefor Eye Tracking and Eye-Blink Detection and Measurement. IEEETransactions on Consumer Electronics, August 2008, incorporated byreference. A detailed AAM lip model, including a hue filtering isproposed for the lips area at I. Bacivarov, M. C. Ionita, and P.Corcoran, A Combined Approach to Feature Extraction for MouthCharacterization and Tracking, in ISSC, Galway, Ireland, 2008,incorporated by reference. Each of these independent models may be usedalone or in combination for expression recognition.

On average, in certain embodiments, the scores obtained for the eyesshape represent 70% of the information contained by the entire faceshape and the mouth is also an important emotion feature carrier forcertain emotions, e.g., surprise, with around 60% independentcontribution in certain embodiments. Further, when combined andprocessed together emotion decoding accuracies increase.

Component-Based AAM Representation of Expressivity

Component-based AAM, as described for example at Zhang and F. S. CohenComponent-based Active Appearance Models for face Modelling, inInternational Conference of Advances in Biometrics, ICB, Hong Kong,China, Jan. 5-7, 2006, incorporated by reference) is an approach thatbenefits from both the generality of a global AAM model and the localoptimizations provided by its sub-models. In addition to a global model,component models of the mouth and two eye-models and/or other featuresmay be used alone or in combination.

In summary, a component-based algorithm may be used in certainembodiments as follows. Two sub-models are built, one for the eye regionand one for the lips. The eye sub-model is then derived in a left and,respectively, a right eye model. At each iteration, the sub-models areinferred from the global model. Their optimums are detected by an AAMfitting procedure. Then, the fitted sub-models are projected back intothe global model.

Improved Component Eye Models Extension of the Eye Model

An initial eye model may be based in certain embodiments on a standardformulation of AAM, e.g., as in T. F. Cootes, G. J. Edwards, and C. J.Taylor, “Active appearance models”, Lecture Notes in Computer Science,vol. 1407, pp. 484-, 1998, incorporated by reference. The model may usea detailed analysis of the eye region in terms of degree of eyelidopening, position of the iris, and/or shape and/or texture of the eye.The model may be designed to be robust to small pose variation. Blinkand gaze actions may also be modelled as in I. Bacivarov, M. Ionita, P.Corcoran, Statistical Models of Appearance for Eye Tracking andEye-Blink Detection and Measurement. IEEE Transactions on ConsumerElectronics, August 2008 and/or Ioana Barcivarov, “Advances in themodeling of Facial Subregions and Facial Expression using ActiveAppearance Modeling Techniques”, PhD Thesis, National University ofIreland Galway, June 2009, each being incorporated by reference.

Other expression conditions that may be taken into account in certainembodiments include head pose, occlusions of one or more components ofthe eye model, and/or differences in expression between the two eyessuch as “winking” where one eye is open, and the other closed. Incertain embodiments, an advantageous model is not constrained to thevariations learned during the training phase. For example, this permitsthe two eyes to deform together and/or independently.

A visual explanation is provided by FIG. 3 which presents two types ofchallenges: in-plane-head-rotation and independent actions of the eyes,i.e., winking. FIG. 3 illustrates an example of poor fitting for winkingin-plane-rotated eyes. This example is advantageously included withother examples including two eyes open or closed in certain embodiments.

A component-based AAM, as described for example at M. D. Cordea, E. M.Petriu, T. E. Whalen, “A 3D-anthropometric-muscle-based activeappearance model, in IEEE Symposium on Virtual Environments”,Human-Computer Interfaces and Measurement Systems, (VECIMS), pp. 88-93,2004 and/or H. Choi, S. Oh, “Real-time Recognition of Facial Expressionusing Active Appearance Model with Second Order Minimization and NeuralNetwork”, International IEEE Conference on Systems, Man and Cybernetics,SMC 06, vol. 2, pp. 1559-1564, is used in certain advantageousembodiments that do not constrain model components to global variations.A face model may combine a global model with a series of sub-models.This approach benefits from both the generality of a global model andthe local optimizations provided by its sub-models. This approach isadapted in certain embodiments to independently model the eye regionswithin a face. Two advantageous example versions of the component-basedmodel adapted for the eye region are described below, particularly forapplication of facial expression recognition.

Component-Based AAM Formulations

In a first stage, a component-based AAM is provided for the eye-regionusing two different approaches. For the first approach, the open-eye andclosed-eye states are modelled separately. For the second approach, eacheye is modelled separately, retaining the mixed (overlapping points)open/closed information for each eye region.

Separation of Open and Closed Eye States

In our first approach, the two-eye appearance for open or closed eyes isused. Mixing open and closed eye shapes or textures can introduce errorsinto the model, and so care is taken here in developing the model.First, a global model uses both eyes open and then both eyes closed. Themodel is refined with two sub-models modelling only some particularfeatures of both eyes, meticulously annotated as shown in FIG. 4. Afirst sub-model represents components of open eyes, i.e. inner eyelidand iris. A second sub-model represents components of closed eyes, i.e.,inner eyelid and outer eyelid.

In the case of closed eyes, the inner eyelid is not composed byoverlapped points, but represented as a straight line. The eyebrows maybe included in the global model for better eye location. They areconsidered generally superfluous for local modelling stages, as theeyebrows may be mostly used for accurate eye location. In these stages,the eye location is believed accurate, coming from the initial globalstage.

The fitting process is represented in FIG. 5. Firstly, the global AAM ismatched 22 which gives a rough eye modeling. Then, a blink detector isapplied 24, determining if the eyes are open or closed. Next, thecorresponding sub-model is applied, e.g., the closed-eyes sub-model 26may be applied if the blink detector 24 indicates a blink, and otherwisethe open-eyes sub-model 28 may be applied. This local sub-model providesa more accurate match to the eye-region.

Only one of the open or closed sub-models is typically used in thisfitting process at 30 or 32, saving computational time. A new global AAMmay be applied at 34, along with an error check at 36 before the processis stopped 38. Another advantage is the accuracy of the shapeannotation, as closed-eye shape is no longer obtained from open-eyeshape. In consequence, fewer errors are introduced in the appearancemodel. The blinking information is still extracted thanks to the globalmodel, but the accuracy of the shape is refined by the relevantsub-model.

Independent Modeling of the Left and the Right Eyes

In a second approach, each eye is modelled independently. A two-eyeglobal model may be used for an accurate location and initial modeling.Then, the matching of each of the two eyes is performed independently.

A global model is created using both eyes open and closed. Then aseparate sub-model is created describing a single open or closed eye andthe different variations in between the two states. The two models,i.e., one global model and one sub-model, are trained and areindependently generated using the same procedures described above.

One valuable aspect of the eye-model is the symmetry between the twoeyes. This characteristic permits, starting from one eye model, e.g.left eye, to auto-generate an equivalent model for the other eye, e.g.,right eye, respectively, simply by mirroring the data. An additionaladvantage is reduced memory space requirements, as only one set ofeye-model characteristics is stored. This can be very important whenconsidering the implementation of such techniques in low-cost consumerelectronic devices such as gaming peripherals.

An advantage of this modelling approach is that it permits that the twoeyes find their optimal position and shape independently. There aresituations, especially when dealing with large pose variations, planerotations, occlusions, or strong facial expressions, when the 2-Dprojection of the eyes loses the property of global symmetry. An exampleof such a situation was presented in FIG. 3, where poor fitting of theglobal AAM was illustrated.

FIGS. 6 a-6 c illustrate a fitting algorithm adapted for thiscomponent-based version in accordance with certain embodiments. FIGS. 6a-6 c illustrate examples of annotation for the global model, for thelocal sub-model and its mirroring in order to obtain the right eye,respectively. Initially the global AAM is fitted, roughly locating thetwo eyes. The points of the global model which correspond to thesub-models are firstly extracted to form the two shape vectors,providing a good initialization for each sub-model. At each iteration,the sub-model is then inferred from the global model, its optimum isdetected by an AAM fitting procedure, and the fitted sub-model isprojected back into the global model providing a refined initializationfor that model. Another projection of the global shape vector onto itsprincipal components space may be used in certain embodiments. This stepmay be used to constrain the two independently obtained eyes such thatthey remain within the limits specified by the global model. In the laststep of the fitting process, the fitting error for this refined globalmodel is compared with the original global model fitting error and adecision is taken to use the global model with the least error. Thisprocess can be repeated until a convergence criteria for the global andlocal errors is achieved for each of the models within the component AAMframework. A single step process may be used that will generally achievesufficient convergence if a Viola-Jones face detector has been used toprovide the first initialization for the global model. A detailedprocess flowchart is provided in FIG. 8 and described in more detailbelow.

In FIGS. 7 a-7 c, some comparative examples are illustrated of fitting.FIG. 7 a, in the first column, shows the effects of fitting a standard,holistic AAM eye model. FIG. 7 b, in the middle column, illustrates theeffect of fitting the independent-eyes sub-models without applyingconstraints from the global model. FIG. 7 c, in the right column, showsthe results using the independent-eyes models combined with constraintsfrom the holistic model.

Comparison of Proposed Component-AAM Approaches

Two different versions of adapting a component-based AAM for the eyeregion are described. The first version locally models both eyes,separating the open and closed eye situations. The second versionindependently models each eye, but it simultaneously includes open andclosed eyes. Both versions may be trained using the same training set asfor the standard eye model. A dataset of 70 pictures, e.g., may becarefully selected to provide a high degree of variability in terms ofsubject individuality, pose, illumination, and facial features. Samplesof the training set are illustrated at FIGS. 7 a-7 c, and see theappendix of Ioana Barcivarov, “Advances in the modeling of FacialSubregions and Facial Expression using Active Appearance ModelingTechniques”, PhD Thesis, National University of Ireland Galway, June2009, incorporated by reference above.

FIG. 8 illustrates a fitting algorithm for component based AAM eyemodel. First, detected and/or tracked face regions or face features areinput at 42. A global AAM model is fit at 44. A projection is made ontoan original face region at 46. The left eye is sub-modelled at 48 andthe right eye is sub-modelled at 50. A common eye sub-model may be usedas illustrated at 52 and input to the left and right eye models 48 and50, respectively. Then the right and left sub-models are fit and backprojected onto the original face region at 54 and 56, respectively. Anew global model is fit at 58. An error determined at either globalmodel fit 44 or 58 may be analyzed at 60. When an error is determined,the sub-model refined global model may be applied at 62, and if not,then the original global model may be applied.

The version of the component-based AAM which uses independent left andright eye-models proved in tests to be more useful and effective acrossa range of applications. In this version, local sub-models used weresingle-eye models, adapted for open and closed eyes. This approachproved capable of accurately determining the eye parameters,particularly in cases of facial expression, where the two eyes deformindependently. It is especially robust to pose variations and toin-plane rotations.

The first version depends more particularly on the accuracy of blinkdetection. FIG. 9 a illustrates the blink detector failing by indicatingopen eyes when the eye are in fact closed. This can cause a failure ofthe algorithm, as the open eye sub-model is chosen for fitting. Not onlydoes it attempt to match the closed eyes with the open-eye model but theglobal alignment of the eye-region is lost.

Even if the global model fails by matching open eyes, the sub-models canstill correctly match the eye image with closed eyes because each localmodel contains both open and closed-eye training data. This situation isrepresented in FIG. 9 b. The component-based eye AAM independentlymodeling the left and right eye, is particularly advantageous whenperforming expression recognition as set below.

A Direct Quantitative Comparison of Modeling Techniques

After visually inspecting the results, a quantitative evaluation of theproposed model is performed on representative examples of the three testsets. The quantitative evaluation of a model performance is realized interms of boundary errors, calculated as Point-to-Point (Pt-Pt) shapeerror, calculated as the Euclidean distance between the ground-truthshape vector and the converged shape vector inside the image frame:

$\begin{matrix}{{{Pt} - {Pt}} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}{\left( {\left( {x_{i} - x_{i}^{g}} \right)^{2} + \left( {y_{i} - y_{i}^{g}} \right)^{2}} \right)^{1/2}.}}}} & (3)\end{matrix}$

where the index g marks the ground truth data, obtained by handannotation.

Another type of error can be calculated, namely the Point-to-Curve shapeerror. It is calculated as the Euclidian norm of the vector of distancesfrom each landmark point of the exact shape to the closest point on theassociated border of the optimized model shape in the image frame:

$\begin{matrix}{{{{Pt} - {Crv}} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}{\min\left( {\left( {x_{i} - {r_{x}^{g}(t)}} \right)^{2} + \left( {y_{i} - {r_{y}^{g}(t)}} \right)^{2}} \right)}^{1/2}}}},} & (4)\end{matrix}$

The mean and standard deviation of Pt-Pt and Pt-Cry are used to evaluatethe boundary errors over a whole set of images. FIG. 10 shows plotsillustrating a histogram of Pt-Pt shape errors calculated with respectto manual ground-truth annotations. The ground-truth annotationsrepresent hand-annotated shapes. In these tests, the standard AAMformulation, the component-based method, when omitting its last stage,i.e., the global fitting, and the component-based AAM with globalfitting are illustrated in the plots. The initialization provided fromthe face detection step is used as benchmark.

FIG. 10 illustrates that the boundary errors for the tested fittingalgorithms are concentrated within lower values as compared to theinitial point generated from the detection algorithm, showing animprovement for eye location. Furthermore, the shape boundary errors areconcentrated within lowest values, indicating that the fullcomponent-based AAM performs advantageously in terms of fittingaccuracy, thus resulting in a clear improvement over the initialposition, as well as over the other fitting algorithms. The advantagesof using a component-based initialization and fitting is mirrored in thehigher accuracies obtained for eye tracking, blink, or gaze detection.

Improved Lips Model

Lips models have been developed with advantageous results based on astandard AAM formulation (see, e.g., I. Bacivarov, M. C. Ionita, and P.Corcoran, A Combined Approach to Feature Extraction for MouthCharacterization and Tracking, in ISSC, Galway, Ireland, 2008, and IoanaBarcivarov, “Advances in the modeling of Facial Subregions and FacialExpression using Active Appearance Modeling Techniques”, PhD Thesis,National University of Ireland Galway, June 2009, incorporated byreference). A weak contrast between the color of the lips and thesurrounding skin can provide a challenge, as there can be significantoverlap in color ranges between the lips and skin regions of the face(see N. Eveno, A. Caplier, P. Y. Coulon, New color transformation forlips segmentation, Proceedings of IEEE Fourth Workshop on MultimediaSignal Processing, October 2001, Cannes, France, pp. 3-8, incorporatedby reference). An improved version of an AAM lips model is provided inaccordance with these embodiments. The standard AAM formulation isimproved by applying a pre-processing step that offers a more accurateinitialization of the lip region. The overall approach, embracing theinitialization and the AAM modeling, is described below. The performanceof the lip model has been tested, including developing two consumerapplications: a lip tracker and a smile detector.

Initializaiton of the Lip Region by Chrominance Analysis

The lips model works advantageously well when a strong initialization isprovided in order to achieve an accurate fitting to unseen images.Consequently, a pre-processing method is provided in accordance withthese embodiments that provides such a robust initialization. Keyinformation related to lips is their red color, although red varies withrespect to individuals, make-up, illumination etc. Therefore, byfiltering the red lip color from the face region, the system is betterable to identify the global shape of the lip region. This approach isbased on the work of Pantie et al. (see, e.g., M. Pantic, L. J. M.Rothkrantz, “Automatic Analysis of Facial Expressions: The State of theArt”, IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 22, no. 12, 2000, pp 1424-1445; M. Pantic, M. Tome, L J. M.Rothkrantz, “A Hybrid approach to mouth features detection”, inProceeding of the 2001 Systems, Man and Cybernetics Conference, 2001,pp. 1188-1193; and M. Pantic, M. F. Valstar, R. Rademaker, L. Maat,Web-based database for facial expression analysis, IEEE InternationalConference on Multimedia and Expo (ICME'05), http://www.mmifacedb.com,2005, each being incorporated by reference), and is adapted for our AAMmodels. Firstly, the input image is transformed into the HSV colourspace, as hue representation is less affected by variations inillumination. This colour space brings invariance to shadows, shading,and highlights and it permits using only the hue component forsegmentation. Then the object of interest is filtered into the reddomain, by applying the following hue filter (see S. Chindaro, F.Deravi, “Directional Properties of Colour Co-occurrence Features for LipLocation and Segmentation”, Proceedings of the 3rd InternationalConference on Audio and Video-Based Biometric Person Authentication, pp.84-89, 2001):

$\begin{matrix}{{f(h)} = \left\{ \begin{matrix}\frac{1 - \left( {h - h_{0}} \right)^{2}}{w^{2}} & {{{h - h_{0}}} \leq w} \\0 & {{{h - h_{0}}} > w}\end{matrix} \right.} & (5)\end{matrix}$where h is the shifted hue value of each pixel so that h₀=⅓ for redcolor. Note that h₀ controls the positioning of the filter in the huecolor plane and w controls the color range of the filter around its h₀value. As the color for the lip region varies with respect to personidentity, light conditions, make-up, etc., the challenge is to findoptimal parameters for the filter. Although an optimal solution would bean adaptive hue filter, the simplified solution adopted in our case isto find the optimal parameters for predefined conditions, e.g., for aspecific database.

A statistical analysis was performed on our training set to study thevariation of lip colour between individuals and the differences causedby varying illumination on lip pixels for the each picture wasinvestigated. Standard deviation does not vary much from picture topicture, as the pictures belong to the same database, with controlledacquisition conditions. The filter coefficients are chosen afterperforming an average on the mean and on the standard deviation, for allpictures. The overall mean is 0.01 and it corresponds to the positioningof the filter h₀. The overall standard deviation approximating thefilter width w is 0.007.

After determining the parameters of the filter and performing the actualfiltering operation, each image may be binarized using a predefinedthreshold, as illustrated at FIGS. 11 a-11 c, which show lip regionpre-processing of original image, a representation achieved after thehue filter, and a further representation achieved after binarization.The value of this threshold was set to 0.5, determined after atrial-and-error testing. Morphological operations such as closing,followed by opening, can be used in order to fill in gaps and toeliminate pixels that do not belong to the lip region. After the lipregion is determined, its center of gravity (COG) is calculated. Thispoint is then used as the initialization point for the AAM fittingalgorithm.

Formulation of an AAM Lip Model

A lips modelling in accordance with certain embodiments is composed oftwo main steps: an initialization step and a modeling step. Referring toFIG. 12, an image is acquired 72. Before the lip feature can beextracted and analysed, a face is detected 74 in the acquired image andits features are traced. The face is inferred from the Viola-Jones facedetector applied on the input image. Then, a region of interest (ROI),i.e. the lip region, is deducted 76 from the rectangle describing thesurroundings of the face. Thus the ROI is reduced to the lower third onthe y axis, while ⅗ of the face box is retained on the x axis, asillustrated at box 78 of FIG. 12. A hue filter is then used to providean initial location of the lip region within this ROI, as illustrated bycorrespondence in the red domain 80 and other processing methods 82.

In a further step, AAM is applied at 84 and 86 in order to perform arefined detection and to determine detailed lips features. The startingpoint for the algorithm is the COG of the hue filtered ROI. The AAMadjusts the parameters so that a synthetic example is generated, whichmatches the image as closely as possible, as seen in FIG. 12. Optimaltexture and shape parameters are determined using the standard AAM lipmodel. In consequence information regarding lip features, such as itscurvature, degree of opening, or its texture, can be readily determined.Extraction and interpretation of mouth features 88 is thereby providedin accordance with certain embodiments.

The Full Component-AAM Face Model

It is impractical to include all possible variations of shape andtexture in a training set, particularly with limited processingresources such as with an embedded device. The overall degrees offreedom inherent in the model are restricted by the number of modelparameters. That is, a model which did incorporate practically allpossible facial variations in its training set would have acorrespondingly large set of model parameters making it unwieldy andimpractical, particularly for applications in consumer electronics orfor implementations in low-cost gaming peripherals. Thus any practicalmodel would restrict its potential variations. To achieve realimprovements, more than a single holistic AAM model is provided inaccordance with certain embodiments.

The component-based AAM may be adapted for facial expression analysis.This approach benefits from both the generality of a global AAM and theoptimizations provided by local sub-models. It adds degrees of freedomto the global model, by accounting for individual variations of facialfeatures.

FIG. 13 illustrates a full component-AAM Face Model incorporating bothimproved Eyes and Lips models in a single component-AAM framework. Adetected and/or tracked face region is input at 92. A global AAM modelis fit at 94. A projection onto an original face region is provided at96. Left eye, right eye and mouth AAM sub-models are applied at 98, 100and 102, respectively. A common eye AAM sub-model may be used with theleft and right eye sub-models 98 and 100, respectively, as describedabove. Each sub-model 98, 100, and 102 is then fit and back projectedonto the original face region at 104. A new global AAM model is fit at106. If there are errors at the global AAM model fit steps 94 or 106,then it is determined at 108 to apply the component based AAM model 110and if not, then to apply the original global AAM model 112.

Now FIG. 13 explains the further adaptation for the entire face region,using two sub-models: both an eye model and a tips model. As for the eyeregion model, at each iteration, the relevant sub-models are inferredfrom the global model. Optimal fittings are determined through an AAMfitting procedure, based solely on shape parameters. Then, the fittedsub-models are projected back into the global model.

FIGS. 14 a-14 c illustrate one practical example of the benefits of acomponent-based representation, on an image containing both pose andexpression variations. This shows how the sub-models improve fitting ofthe global model to the entire face region, which in turn improves thealignment of each sub-model with the local features which are importantto accurate expression recognition. FIG. 14 a is a result of aconventional AAM. FIG. 14 b represents the fitting of the AAMsub-models. FIG. 14 c depicts the component-based result.

Now, in order to quantitatively evaluate the overall performances of acomponent-based AAM, its accuracy is measured in terms of expressionclassification/recognition rates. First, consider which AAM features aremost relevant for expression analysis.

Relevant Feature Extraction for Facial Expressions

Feature selection involves keeping the most relevant features forclassification and discarding irrelevant or redundant features. Thequality of the extracted features plays a key role in theirclassification. Two types of features can be extracted when applying anAAM: geometric features and features depicting the texture. Optionally,appearance parameters can be obtained by applying PCA on theconcatenated geometric and texture parameters. Information regardingboth types of features are valuable when fitting an AAM model to unseenimages. Now it is determined which of these parameters are moresignificant for determining and recognizing facial expressions.

Features for Facial Expression Recognition (FER)

Shape features have a large role to play in facial expressionrecognition (see, e.g Lucey, S., A. B. Ashraf, and J. Cohn,“Investigating Spontaneous Facial Action Recognition through AAMRepresentations of the Face”, Face Recognition Book, edited by K.Kurihara, ProLiteratur Verlag, Mammendorf, Germany, 2007, and Zalewski,L. and S. Gong. “2D statistical models of facial expressions forrealistic 3D avatar animation”, in Computer Vision and PatternRecognition, CVPR, 20-25 Jun. 2005, each being incorporated byreference). It may be advantageous to have both shape and texturefeatures to achieve results (Kotsia, I., et al. “Texture and ShapeInformation Fusion for Facial Action Unit Recognition”, in FirstInternational Conference on Advances in Computer-Human Interaction(ACHI), 2008, incorporated by reference).

On one hand, the skin texture of the facial region exhibits only slightchanges during facial expressions. These minor changes arise due tolocal lighting effects, blushing, or wrinkling of the skin. Such changescan be considered relatively invariant when compared with the moreobvious changes of shape observed for different facial expressions. Suchgeometrical features are very directly related to expressions. Asexamples, when surprised, we widely open the eyes and eventually themouth, resulting in an elongated chin; when sad, we often blink; whenangry, the eyebrows are usually drawn together. Thus, shape parametersare typically the most significant features in facial expressiondecoding.

Quantitative Determination of AAM Parameters for FER

While shape parameters of an AAM face model contain the greatest“information density” for determining facial expression, there can tendto be a significant amount of redundant information within thisparameter set. Contra-wise, there may also be significant usefulinformation contained in a small subset of the texture parameters.Advantageously, we have further refined our use of AAM model parametersto achieve a more optimal set of parameters which are more closely tunedto facial expression analysis.

The particular shape parameters that best represent facial expressionswere investigated. A customized database and images from the FEEDTUMdatabase were used. A set of 42 images were used to train the improvedcomponent model, 21 from each database. This training set was picked toinclude images of many different individuals, with varying facial poses,and facial expressions. Corresponding recognition tests were performedon a set of 50 unseen images from a database and 100 from FEEDTUM. AEuclidean NN based classifier was used for measuring each category offacial expression and, after discarding the lowest order shape parameterwhich is dominated by the facial pose information, the 17 lowest ordershape parameters of the AAM model were used as features.

FIG. 15 show plots that illustrate means of shape parameters over eachof the seven universal emotions. FIG. 15 includes plots of expressionrecognition rates versus numbers of model parameters employed. Thisillustrates the mean over the AAM shape parameters for each of the sixuniversal facial expressions and the neutral one of the training set.

A maximum of 17 shape parameters is used in total in accordance withcertain embodiments. Then, the number of parameters is reduced,eliminating in turn the parameter with the smallest variation across thetest set of images. As the number of parameters is reduced, what areleft are the parameters which exhibit the widest variation. Theseresults are presented in the plot of FIG. 16, where FER accuracy isplotted against the number of shape parameters used in that particulartest.

Optimal results are obtained in accordance with certain embodiments whenthe five to six most variable parameters are used. As the number ofparameters is increased beyond the 6^(th) parameter, the model accuracydeteriorates. An educated choice of parameters positively affects thesystem performances. It is shown empirically that using the first 30-40%of model parameters provides higher expression recognition rates.

A Note on the Lowest-Order Shape Parameter

After performing PCA on shape parameters, the information onout-of-plane head pose was encoded in the first shape parameter. This isexplained by the fact that the variations caused by pose causesignificantly more geometric distortion than the variation caused bydifferences between individuals or between facial expressions.Consequently, pose variation is uncorrelated to a large extent withother sources and manifests itself in the first-order PCA parameter.

FIG. 17 shows bar graphs that illustrate different FER classificationschemes comparing recognition rates with and without the lowest orderAAM shape parameter. The pose parameters are advantageous for the AAMfitting stage and the subsequent extraction of facial features. The posevariation information contributes to the shape and texture of facialfeatures. However when analyzing facial expressions, the lowest orderpose parameter should generally be eliminated, as it will not containthe required facial expression information. This is illustrated in FIG.17, where a range of FER classification methods are compared each withand without the first-order AAM pose parameter. SVM based methods aremore robust than NN methods to pose. Even where the effect of this poseparameter is negligible, it still adds an additional computationalburden which is redundant as even for SVM the recognition rates areslightly lower with this parameter included. This would not be the caseif the model is trained using only frontal images, but then the modelwould not be able to generalize to non-frontal faces.

Conclusions on the Significance of AAM Features

Based on the results of these tests, shape parameters did indeed proveto be overall the most valuable features for facial expression decodingaccuracy. Other tests confirmed that while shape results are comparablewith the results obtained when applying a combined shape and texturemodel, i.e., when using the AAM appearance features, the number of shapeparameters is significantly less. In turn the computational requirementsboth for feature extraction and to subsequently perform classificationare also reduced. Thus, shape parameters on their own have ademonstrable advantage over approaches based on texture-only and bothconcatenated and merged shape and texture. These finding were alsoconfirmed by the authors Zalewski, L. and S. Gong, see “2D statisticalmodels of facial expressions for realistic 3D avatar animation”, inComputer Vision and Pattern Recognition, CVPR. 20-25 Jun. 2005,incorporated by reference. The accuracy rates of classification were notspecifically addressed in this series of experiments. Improvements aredescribed below to increase the accuracy of expression classifiers.

Expression Classification and Recognition

The last step in a facial expression recognition (FER) system isexpression classification and recognition. Two classifiers may becompared: SVM and NN (see Gualtieri, J. A. and R. F. Cromp. Supportvector machines for hyperspectral remote sensing classification, in 27thAIPR Workshop: Advances in Computer-Assisted Recognition. 1998.Washington, D.C.: SPIE, incorporated by reference). In accordance withcertain embodiments, the classifiers use as input relevant AAMparameters and they present at output the choice between two facialexpressions. When dealing with poses, the pose parameters may bediscarded.

Defining a Fixed Set of Classes for Facial Expression

Facial Action Coding System (FACS), originally developed by Ekman andFriesen in 1976 (see, Ekman, P. and W. Friesen, “Facial Action CodingSystem: A Technique for the Measurement of Facial Movemen”, ConsultingPsychologists Press, Palo Alto, 1976, incorporated by reference) is themost widely used coding system in the behavioural sciences. The systemwas originally developed by analysing video footage of a range ofindividuals and associating facial appearance changes with contractionsof the underlying muscles. The outcome was an encoding of 44 distinctaction units (AUs), i.e., anatomically related to contraction ofspecific facial muscles, each of which is intrinsically related to asmall set of localized muscular activations. Using FACS, one canmanually code nearly any anatomically possible facial expression,decomposing it into the specific AUs and their temporal segments thatproduced the expression. Resulting expressions can be described usingthe 44 AUs described by Ekman or combinations of the 44 AUs. In 2002, anew version of FACS was published, with large contributions by JosephHager (see, J. Hager, P. Ekman, and W. Friesen, “Facial action codingsystem”, Salt Lake City, Utah: A Human Face, 2002, incorporated byreference).

Ekman and Friesen have also postulated six primary emotions which theyconsider to be universal across human ethnicities and cultures. Thesesix universal emotions, commonly referred as basic emotions are:happiness, anger, surprise, disgust, fear, and sadness, illustrated inFIG. 18, which also includes the neutral state in the third row from thetop. The six basic emotions and the neutral state are expressed in FIG.18 by different subjects. In order from top row to bottom row, FIG. 18illustrates anger, happiness, neutral, surprise, fear, sadness, anddisgust. The leading study of Ekman and Friesen formed the origin offacial expression analysis, when the authors proclaimed that the sixbasic prototypical facial expressions are recognised universally. Mostresearchers argue that these expressions categories are not sufficientto describe all facial expressions in detail. The neutral expression, ora face without expression, is included as a seventh category inaccordance with certain embodiments.

Expression Classification

The choice of the two classifiers is based on their positive resultsobtained in the literature. E.g., in T. F. Cootes, G. J. Edwards, and C.J. Taylor, “Active appearance models”, Lecture Notes in ComputerScience, vol. 1407, pp. 484-, 1998, incorporated by reference, it isstated and it is proved by experiments for gender classification andface recognition that SVM is typically among the top two classifiers,and the other top ranking classifier is one of the Euclidean-NN rule orthe cosine-NN rule.

Nearest Neighbour (NN)-Based Classifier

Some experimental results are provided below with variations on the NNtechnique to determine its practical utility. In particular, severaldifferent types of similarity metric and templates for NN wereinvestigated, including the Euclidean, cosine, and Mahalanobisdistances. Also, two types of templates can be employed, based oncalculating the mean or the median over the input AAM parameters.

For these experiments, each picture in the test set is classified bycomparing its parameters with a template corresponding to eachexpression. The class yielding the minimum distance is selected. Thetemplates for each class are obtained by a mean, or a median, over theshape parameter vector for each expression. The classifiers are of typeexpression 1/expression 2, i.e., neutral/non-neutral, sad/non-sad,angry/non-angry, disgusted/non-disgusted, fear/non-fear,surprised/non-surprised, and happy/non-happy, and of typeexpression/non-expression. Considering that there are six expressionsand the neutral one, then 28 classifiers are obtained, seven for theformer type and 21 for the latter.

For a first set of experiments, the AAM training and test sets coincide,i.e., there are no unseen images. This is so that the results are notbiased with poor AAM fittings. The Tables NN1-NN4 summarise theclassification rates, as for example, in Table 8.2: the system 80%correctly recognises angry from non-angry faces and 75% angry fromdisgusted faces.

TABLE NN1 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE MMI DATABASE(TRAINING AND TEST SETS OVERLAP) WHEN USING A NN WITH A MEAN TEMPLATERULE-AVERAGE OF EXPRESSION CLASSIFICATION 83.67%. A D F H N Sa Su A 8075 85 90 75 80 87.5 D 87.5 85 95 92.5 95 95 F 57.5 85 67.5 72.5 82.5 H87.5 85 95 95 N 80 70 85 Sa 85 87.5 Su 85

TABLE NN2 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE MMI DATABASE(TRAINING AND TEST SETS OVERLAP) WHEN USING A NN WITH A MEDIAN TEMPLATERULE-AVERAGE OF EXPRESSION CLASSIFICATION 83.41%. A D F H N Sa Su A 82.590 72.5 77.5 77.5 90 85 D 85 90 82.5 90 97.5 92.5 F 67.5 75 75 85 82.5 H77.5 75 87.5 90 N 90 72.5 85 Sa 87.5 88 Su 85

TABLE NN3 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE FEEDTUMDATABASE (TRAINING AND TEST SETS OVERLAP) WHEN USING A NN WITH A WITH AMEAN TEMPLATE RULE-AVERAGE OF EXPRESSION CLASSIFICATION 93.58%. A D F HN Sa Su A 100 96.7 93.4 100 96.7 96.7 100 D 80 90 100 70 90 100 F 83.496.7 80 93.4 100 H 90 90 100 100 N 100 93.4 100 Sa 80 100 Su 100

TABLE NN4 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE FEEDTUMDATABASE (TRAINING AND TEST SETS OVERLAP) WHEN USING A NN WITH A MEDIANTEMPLATE RULE- AVERAGE OF EXPRESSION CLASSIFICATION 89.79%. A D F H N SaSu A 100 63.4 93.4 100 96.7 90 100 D 73.4 86.7 96.7 70 93.4 100 F 73.496.6 66.7 90 93.4 H 90 90 96.7 100 N 83.4 86.7 100 Sa 83.4 100 Su 100

Generalization of the AAM to unseen subjects is tested in a secondseries of experiments with “leave-one-out” tests, in which images of thetested subject are excluded from training. A decrease in theclassification accuracies is observed. Tables NN5 and NN6 summarize thecorresponding results of these tests. The classifier performs better onthe FEEDTUM database. The MMI presents more variations in illumination,which is a condition that affects the AAM fitting and, in consequence,the precisions of facial feature extraction, particularly for unseenimages.

TABLE NN5 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE MMI DATABASE(TRAINING AND TEST SETS DO NOT OVERLAP) WHEN USING A NN WITH A MEANTEMPLATE RULE-AVERAGE OF EXPRESSION CLASSIFICATION 62.99%. A D F H N SaSu A 83.33 50 63.4 80 50 63.4 76.7 D 60 70 66.7 56.7 53.4 73.4 F 56.783.4 56.7 80 63.4 H 70 73.4 53.4 80 N 60 56.7 83.4 Sa 60 86.7 Su 63.4

TABLE NN6 EXPRESSION CLASSIFICATION ACCURACIES (%) FOR THE FEEDTUMDATABASE (TRAINING AND TEST SETS DO NOT OVERLAP) WHEN USING A NN WITH AMEAN TEMPLATE RULE-AVERAGE OF EXPRESSION CLASSIFICATION 66.94%. A D F HN Sa Su A 83.33 50 63.4 80 50 63.4 76.7 D 60 70 66.7 56.7 53.4 73.4 F56.7 83.4 56.7 80 63.4 H 70 73.4 53.4 80 N 60 56.7 83.4 Sa 60 86.7 Su63.4

A template obtained by averaging the AAM shape parameters outperforms,slightly, a template based on the median approach. Thus in theexperiments that follow, a template obtained by averaging the shapeparameters is used for NN. From these experiments on NN-based expressionclassification, it has been determined that Euclidean and cosinedistances perform almost equally well, but there is a slight advantagefor the Euclidean distance from the point of view of a more consistentlevel of performance. Thus in accordance with certain embodiments, theEuclidean distance is used as a representative of the optimal NNtechnique.

Support Vector Machines (SVM) Classifier

Experiments on SVM classifiers were also performed to determine theoptimal settings for SVM when applied to expression classification. Herewe search for the best kernel function, and also for the optimalsettings for each function. Two potential kernel functions areinvestigated: the residual basis function (RBF) and the polynomialfunction. Best grade for a polynomial kernel and the optimal δ valuesfor an RBF kernel are searched.

In the first part of the experiment, seven SVM classifiers of typeexpression 1/expression 2 are built to distinguish the six universalexpressions and the neutral one. The polynomial kernel, with order from1 to 6 and the RBF kernel with δ from 2⁻² to 2⁶ results areinvestigated. The results for the MMI database are detailed in TableSVM1. The optimal kernel function proved to be RBF, with δ fixed onsmall values. In certain embodiments, RBF is used based on the accuracyand the consistency of its results. In a second series of experiments,we sought to confirm the previous conclusions, this time using 28classifiers, as in our earlier NN trials. The results, presented intables SVM2 and SVM3, confirm that RBF with δ fixed on small valuesachieves the highest classification rates.

The RBF kernel function with δ fixed on small values gave the bestresults. Our findings are confirmed also in the literature, where theRBF kernel is the most common function to be used in expressionrecognition with SVM. The choice is also motivated by theory. RBF hasfewer adjustable parameters than any other commonly used kernel and itthus has less numerical complexity.

TABLE SVMI EXPRESSION CLASSIFICATION USING SVM FOR THE MMI DATABASES,WHEN USING DIFFERENT KERNELS. Polynomial-grade Classifier 1 2 3 4 5 6H/non-H 88.7 78.7 82.7 65.4 59.4 51.4 D/non-D 56 46.7 46 46 11.4 40.7Su/non-Su 74.7 52.7 51.4 50 46 44.7 A/non-A 51.4 47.4 47.4 46 53.4 53.4Sa/non-Sa 74 58 54 47.4 45.4 43.4 F/non-F 78.7 72 71.4 64.7 60.7 54N/non-N 62.7 56 59.4 54.7 50.7 45.5 Average 69.5 58.8 58.9 53.5 51 47.6RBF Classifier 2⁻³ 2⁻² 2⁻¹ 2^(n) 2¹ 2² 2³ H/non-H 78.7 82 82.7 82.7 9092.7 92.7 D/non-D 63.4 58 60.7 55.4 70.7 79.4 62.7 Su/non-Su 40.7 46.761.4 71.4 72 71.4 68.7 A/non-A 71.4 71.4 40 72.7 66 70 48 Sa/non-Sa 6664 61.4 62.7 69.4 68.7 70.7 F/non-F 69.4 69.4 70 70.7 72.7 72 72 N/non-N 38.7 47.7 58.7 62.7 61.4 79.4 59.4 Average 61.2 62.1 62.2 68.4 71.876.3 67.8

TABLE SVM2 ACCURACY (%) OF EXPRESSION CLASSIFICATION ON FEEDTUM FOR THEAAM SHAPE PARAMETERS WHEN APPLYING A SVM CLASSIFIER FOR RBF 2², AVERAGEOF EXPRESSION CLASSIFICATION 69.43%. A D F H N Sa Su A 70 63 70 66.7 7070 73.4 D 79.4 63.4 70 73.4 53.4 70 F 72 66.7 56.7 80 53.4 H 92.7 73.460 66.7 N 79.4 53.4 80 Sa 68.7 76.7 Su 71.4

TABLE SVM3 ACCURACY (%) OF EXPRESSION CLASSIFICATION ON MMI FOR THE AAMSHAPE PARAMETERS WHEN APPLYING A SVM CLASSIFIER FOR RBF 2², AVERAGE OFEXPRESSION CLASSIFICATION 62.59%. A D F H N Sa Su A 62.5 55 65 52.5 52.555 65 D 65 65 52.5 60 62.5 77.5 F 55 62.5 60 52.5 77.5 H 57.5 57.5 65 75N 52.5 57.5 77.5 Sa 57.5 75 Su 80

Expression Recognition

Now, after training a set of classifiers to discriminate between twofacial expressions, we would like to be able to associate a human facewith one of the six universal expressions or the neutral one. Thisprocess is known as facial expression recognition. We will again compareNN and SVM based techniques, but this time adapted for a multi-classdecision framework.

Nearest Neighbour

The first set of experiments was performed on FEEDTUM, MMI and our owndatabase using the Euclidean-NN and the optimal number of 7 shapeparameters. Between 20 and 30 pictures were used for training and 150test images were drawn each from FEEDTUM and MMI with an additional 50from our own database. Table AAM1 summarizes expression recognitionrates when a conventional AAM facial representation is used. The highestrecognition rates are obtained on the MMI database. This is explained bythe fact that MMI has a better image resolution and less variation inillumination than many other datasets. The poorest results are obtainedon databases which contain the strongest pose and illuminationvariations. These results serve mainly to confirm the relatively poorperformance of conventional AAM.

TABLE AAM1 SUMMARY OF THE SYSTEM ACCURACIES (%) FOR RECOGNISING EMOTIONSWITH EUCLIDEAN-NN. Database Recognition rate (%) FEEDTUM 35.71 MMI 40.88Our database 22.66 Overall 33.08

Multi-Class SVM

By their nature SVMs are binary classifiers. However, there existstrategies by which SVMs can be adapted to multi-class tasks, such asOne-Against-One (1A1), One-Against-All (1AA), Directed Acyclic Graph(DAG), and SVMs in cascade structures. In the following experiments weonly exemplify the 1A1 and the cascade structures. The choice is basedon their simplicity and their good results (see Wallhoff, F, “The FacialExpressions and Emotions Database Homepage (FEEDTUM)”,www.mmk.ei.tum.de{tilde over (/)}waf/fgnet/feedtum.html. September 2005;and Wallhoff, F., et al. Efficient Recognition of authentic dynamicfacial expressions on the FEEDTUM database. in IEEE InternationalConference on Multimedia and Expo. 9-12 Jul. 2006, each beingincorporated by reference). The following experiments use the same testinputs as the NN tests presented in table AAM1.

For the first part of this experiment, we employed the 1A1 approach.Altogether 21 classifiers are applied for each picture in ourtest-bench. A general score is calculated for each picture. The“recognized” facial expression is considered to be the one which obtainsthe highest score. As an example, to calculate the score for a “happyface” we apply: happy/fear; happy/sad; happy/surprised; happy/neutral;happy/angry; and happy/disgusted. Every time that a happy face isidentified, a counter is incremented that represents its score. Resultsare summarized in Table AAM2.

TABLE AAM2 EXPRESSION RECOGNITION ACCURACIES FOR 1AA- SVM ON THE FEEDTUMAND MMI DATABASES. Emotion FEEDTUM (%) MMI (%) Surprise 71 76.1 Fear66.3 62.5 Happiness 65.4 62.5 Anger 64.8 56.8 Neutral 62.8 60.4 Sadness61.5 59.7 Disgust 57.6 64 Overall 64.2 63.15

In the second part of the experiment, another alternative to extend thebinary SVM to a multi-class SVM is investigated. A cascaded structureincluding the most effective six classifiers of the seven classifyingthe six universal expressions and the neutral expression are used. Theworkflow and the corresponding results are summarized in FIG. 19.Performances of expression recognition of SVM classifiers are providedin a cascade structure, for MMI and FEEDTUM databases on the left andright in FIG. 19, respectively. FIG. 19 illustrates the recognition rateafter each stage of the cascade, e.g. our system correctly recognizes asurprised face with a probability of 84.5% or an angry face with aprobability of 70%.

Conclusions on Classifier Performances

In this section, we analyzed approaches and corresponding results forexpression classification and recognition in still images. Twoclassifiers, namely SVM and NN, were compared. After performing a seriesof five experiments, we conclude the following:

-   -   Overall, happiness proved to be the most recognizable        expression, followed by surprise. These observations can be        explained by the fact these particular expressions affect the        shape of the facial features more than other expressions, e.g.,        note the open mouth and raised mouth corners. These expressions        are followed in terms of recognisability by anger, sadness,        disgust, neutral state, and fear.    -   The results prove system behaviour is consistent across subjects        of both genders and several races and ages.    -   SVM is more effective than NN as a classifier, both with respect        to higher classification/recognition rates and better        consistency of the results    -   Best results for SVM were obtained when using RBF kernel        function with δ fixed on small values.

Best results for NN were obtained when using the Euclidean distance anda template obtained from averaging the shape parameters as a metric forclassification. These settings are also to be used in our next series ofexperiments.

Results and Applications FER Compared Across Different ModelingStrategies

Towards the end of the previous section we presented results for both NNand SVM FER using a conventional AAM model for feature extraction. Theresults of both approaches were quite disappointing. In this section wepresent a summary of detailed comparisons across different modelingstrategies for both NN and SVM techniques.

TABLE NN FINAL - SYSTEM ACCURACY (%) FOR CLASSIFYING FACIAL EXPRESSIONSUSING EYE-AREA, MOUTH AREA, FACE MODELED WITH A CONVENTIONAL ORCOMPONENT-BASED AAM, WITH EUCLIDEAN-NN FEEDTUM Classifier Eyes Lips FaceComp Surprised/Disg. (%) 44.37 55.87 85.83 83.33 Surprised/Happy (%)28.75 69.37 88.33 83.33 Happy/Sad (%) 41.87 46.25 68.33 82.22Surprised/Sad (%) 33.12 51.66 93.33 83.33 Overall (%) 37.02 55.78 83.9583.05 MMI Classifier Eyes Lips Face Comp Surprised/Disg. (%) 73.12 76.6685.62 78.33 Surprised/Happy (%) 68.75 75 86.25 79.16 Happy/Sad (%) 72.550 73.12 77.5 Surprised/Sad (%) 69.37 52.5 77 73.33 Overall (%) 70.9363.54 80.49 77.08

The expression classification and recognition methods are tested on thespecialized databases FEEDTUM and MMI, while their accuracy against posevariations is tested on pictures collected especially for thisexperiment. The results obtained from our tests, suggest that the systemis robust in dealing with subjects of both genders. Also, it isindependent of race and age.

Here the contributions of each of the AAM sub-models to expressionanalysis are evaluated. The results are compared with the results of aholistic AAM face model and a component-based AAM facial representation.Table NN-Final summarizes the classification results for a mouth model,two-eye model, a conventional AAM face model, and a component-based faceAAM when using NN. In the corresponding Table SVM-Final we showcorresponding results for the SVM classification scheme.

Comparison and Discussion of FER Results

The classification rate average when using a component-based AAM is of73.02%, while for a classical AAM is of 69.43%. The results confirm theimprovement brought by a sub-model representation although these are notquite as impressive as we would have felt from the initial improvementsnoted during our studies on the feature extraction and the classifiersfor individual facial expressions. The SVM is quite good at compensatingfor the deficiencies of the conventional AAM and has reduced thebenefits we would have expected from the use of our improvedcomponent-AAM approach.

TABLE SVM-FINAL-SYSTEM ACCURACIES (%) FOR SVM CLASSIFICATION OF EMOTIONSA CONVENTIONAL(1) AND COMPONENT-BASED(2) AAM. A D F H 1 2 1 2 1 2 1 2 A70 75 63 75 70 75 66.7 68 D 79.4 73.4 63.4 70 70 75 F 72 70 66.7 73.4 H92.7 93.4 N Sa Su N Sa Su 1 2 1 2 1 2 A 70 75 70 79.4 73.4 68.7 D 73.475 53.4 68 70 71.4 F 56.7 68.7 80 83.4 53.4 63 H 73.4 75 60 75 66.7 66.7N 79.4 68.7 53.4 63 80 83.4 Sa 68.7 63.4 76.7 75 Su 71.4 73.4

Application to Computer Gaming Workflows

Our system demonstrates the real-time capability to determine a person'semotions. Such a system suggests a range of applications relevant tocomputer gaming environments, both single and multi-player.

Adaptation of Game Difficulty Based on User Emotions

Most computer games allow a user to select between a number ofdifficulty levels. Typically, this selection is made only once, at thebeginning of a game and cannot be changed without starting a newinstance of the game. A new game workflow is provided in accordance withcertain embodiments where the emotions presented by a user via theirfacial expressions are evaluated in an ongoing basis and, where certaincriteria are met, the difficulty level of a game will be adjustedupwards or downwards accordingly. In particular, the system in certainembodiments uses the angry, disgusted and sad face expressions asnegative indicators suggesting that the current difficulty level ismaking the player unhappy. If a user continues in such negative states,then after a couple of minutes, the system can ease the difficulty levelor provide some hints to achieving the next gameplay goal, or some otheraction to lift the spirits of the player, such as for example adjustingthe foreground, background and/or characteristics of interactive objectslike speed, intensity, potency of potential negative effect on theplayer, and/or adjusting the tone of audio or text and/or improving theperformance of the subject. Contra-wise if the player is showing toomuch neutral face then it is determined in certain embodiments that thegame has become too easy and they are bored. A happy face can beconsidered as an indicator of the correct level of difficulty as theuser is still enjoying the game and presumably succeeding in realizingtheir objectives. These are just examples of ways to adjust gameparameters based on recognized expressions on a player's face.

Adapting Game Workflow from User Responses

In addition to the basic determination of difficulty level based onfacial expression recognition and/or recognized identity of the person'sface and predetermined settings, we have also demonstrated a gradedclassification of several of these emotions in a real-time embodiment(see, e.g., Ioana Barcivarov, “Advances in the modeling of FacialSubregions and Facial Expression using Active Appearance ModelingTechniques”, PhD Thesis, National University of Ireland Galway, June2009, incorporated by reference). When combined, this enables not onlythe classification of these expressions but a measurement of thetransition from a mild to a more intense expression. Such a metricprovides interesting possibilities for adapting the more detailedworkflow and storyline of a game.

In certain further embodiments, we take the use of facial expressions tothe next level suggesting that the actual workflow and storyline of thegaming environment can be adapted according to the emotional state ofthe game player at critical waypoints or challenges along the game path.In conventional gaming, such alterations are only achieved based on theactual outcomes of a challenge in the game environment. Our techniquesprovide means for the game designer to achieve a richer and moredetailed interaction with the players' states of mind before, during andimmediately after critical junctures in the gaming storyline.

Real-Time Personalized Avatars

In a recent computer game, Little Big Planet, the game avatarsassociated with each player can be endowed with rudimentary facialexpressions, e.g., pushing the up-arrow on the gamepad will generate asmiling face; a second press makes the expression even happier and a fewmore button-presses and your avatar will have a very silly grinthroughout the game! Our concept is more challenging. In certainembodiments, dynamic detection of user facial expression is mirrored bytheir in-game avatar in real-time.

In a companion disclosure we describe an enhanced face model derivedfrom active appearance model (AAM) techniques which employs adifferential spatial subspace to provide an enhanced real-time depthmap. Employing techniques from advanced AAM face model generation [31]and the information available from an enhanced depth map we can generatea real-time 3D face model. The next step, based on the 3D face model isto generate a 3D avatar that can mimic the face of a user in real time.We are currently exploring various approaches to implement such a systemusing our real-time stereoscopic imaging system.

While exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention as set forth in the claims that follow and their structuraland functional equivalents.

In addition, in methods that may be performed according to preferred andalternative embodiments and claims herein, the operations have beendescribed in selected typographical sequences. However, the sequenceshave been selected and so ordered for typographical convenience and arenot intended to imply any particular order for performing theoperations, unless a particular ordering is expressly indicated as beingrequired or is understood by those skilled in the art as beingnecessary.

Many references have been cited above herein, and in addition to thatwhich is described as background, the invention summary, briefdescription of the drawings, the drawings and the abstract, thesereferences are hereby incorporated by reference into the detaileddescription of the preferred embodiments, as disclosing alternativeembodiments of elements or features of the embodiments not otherwiseexpressly set forth in detail above. In addition, the following areincorporated by reference as disclosing alternative embodiments: U.S.Pat. Nos. 7,565,030, 7,715,597, 7,515,740, and 7,620,218 and USpublished patent applications nos. 2009/0179998, 2010/0066822,2008/0013798, 2008/0175481, 2009/0263022, 2009/0238419, 2009/0167893,2009/0003661, 2009/0196466, 2009/0080713, 2009/0189998, 2009/0185753,2009/0190803, 2009/0179999, 2007/0201724, 20070201725, 2010/0026833 and2009/0303342, and U.S. patent application Ser. Nos. 12/374,040,12/631,711, 61/182,625, 61/221,455 and 60/221,467.

What is claimed is:
 1. A method of recognizing a facial expression,comprising: using a processor; acquiring a sequence of digital images;detecting a facial region within a first digital image; applying afacial model, comprising both shape and texture components, to saidfacial region to obtain a first detailed match to the facial region;applying a feature model, comprising both shape and texture components,to a feature within the facial region to obtain a first detailed matchto said feature; adjusting the facial model, responsive to the firstdetailed match of the second model to said feature, to obtain animproved match of the facial model to the facial region; adjusting thefeature model, responsive to the improved match of the facial model tosaid facial region, to obtain an improved match to the feature;extracting principal shape components from both the facial model and thefeature model; and characterizing said principal shape components todetermine a best matching facial expression of a discrete set of facialexpressions.
 2. The method of claim 1, further comprising recognizinganother facial expression within one or more additional digital imagesfrom said sequence.
 3. The method of claim 1, wherein the featurecomprises one or two eyes, an eye feature, one or two lips or partiallips, or one or more other mouth features, or combinations thereof. 4.The method of claim 1, wherein the feature comprises a geometricfeature.
 5. The method of claim 4, wherein the geometric featurecomprises a shape, deformation or location of one or more facialcomponents, or a pose variation, or combinations thereof.
 6. The methodof claim 1, wherein the feature comprises an appearance feature.
 7. Themethod of claim 6, wherein the appearance feature comprises a skintexture change.
 8. The method of claim 7, wherein the skin texturechange comprises a furrow, a bulge, an expression wrinkle, or anillumination variation, or blushing, or combinations thereof.
 9. Themethod of claim 1, further comprising categorizing the best matchingfacial expression as indicating surprise, fear, happiness, anger,neutral, sadness, or disgust.
 10. The method of claim 1, furthercomprising adapting a game difficulty based on the best matching facialexpression.
 11. The method of claim 1, further comprising adapting agame workflow based on the best matching facial expression.
 12. Themethod of claim 1, further comprising mirroring the best matching facialexpression in an avatar within a game display.
 13. The method of claim1, wherein the facial model comprises an Active Appearance Model (AAM).14. The method of claim 1, wherein the principal shape componentscomprise seven or less principal shape components.
 15. The method ofclaim 1, wherein the principal shape components comprise six or sevenprincipal shape components.
 16. One or more non-transitory computerreadable media having code embedded therein for programming a processorto perform a method of recognizing a facial expression within a facialregion detected within a digital image acquired among a sequence ofdigital images, wherein the method comprises: applying a facial model,comprising both shape and texture components, to said facial region toobtain a first detailed match to the facial region; applying a featuremodel, comprising both shape and texture components, to a feature withinthe facial region to obtain a first detailed match to said feature;adjusting the facial model, responsive to the first detailed match ofthe second model to said feature, to obtain an improved match of thefacial model to the facial region; adjusting the feature model,responsive to the improved match of the facial model to said facialregion, to obtain an improved match to the feature; extracting principalshape components from both the facial model and the feature model; andcharacterizing said principal shape components to determine a bestmatching facial expression of a discrete set of facial expressions. 17.The one or more non-transitory computer readable media of claim 16,wherein the method further comprises recognizing another facialexpression within one or more additional digital images from saidsequence.
 18. The one or more non-transitory computer readable media ofclaim 16, wherein the feature comprises one or two eyes, an eye feature,one or two lips or partial lips, or one or more other mouth features, orcombinations thereof.
 19. The one or more non-transitory computerreadable media of claim 16, wherein the feature comprises a geometricfeature.
 20. The one or more non-transitory computer readable media ofclaim 19, wherein the geometric feature comprises a shape, deformationor location of one or more facial components, or a pose variation, orcombinations thereof.
 21. The one or more non-transitory computerreadable media of claim 16, wherein the feature comprises an appearancefeature.
 22. The one or more non-transitory computer readable media ofclaim 21, wherein the appearance feature comprises a skin texturechange.
 23. The one or more non-transitory computer readable media ofclaim 22, wherein the skin texture change comprises a furrow, a bulge,an expression wrinkle, or an illumination variation, or blushing, orcombinations thereof.
 24. The one or more non-transitory computerreadable media of claim 16, wherein the method further comprisescategorizing the best matching facial expression as indicating surprise,fear, happiness, anger, neutral, sadness, or disgust.
 25. The one ormore non-transitory computer readable media of claim 16, wherein themethod further comprises adapting a game difficulty based on the bestmatching facial expression.
 26. The one or more non-transitory computerreadable media of claim 16, wherein the method further comprisesadapting a game workflow based on the best matching facial expression.27. The one or more non-transitory computer readable media of claim 16,wherein the method further comprises mirroring the best matching facialexpression in an avatar within a game display.
 28. The one or morenon-transitory computer readable media of claim 16, wherein the facialmodel comprises an Active Appearance Model (AAM).
 29. The one or morenon-transitory computer readable media of claim 16, wherein theprincipal shape components comprise seven or less principal shapecomponents.
 30. The one or more non-transitory computer readable mediaof claim 16, wherein the principal shape components comprise six orseven principal shape components.
 31. A digital image acquisition deviceconfigured to recognize a facial expression, comprising: a lens andimage sensor for capturing a digital image; a processor; a facedetection module configured to detect a facial region within the digitalimage; a feature extraction module configured to program the processorto extract a feature within the facial region a facial model applicationmodule configured to program the processor to apply a facial model,comprising both shape and texture components, to said facial region toobtain a first detailed match to the facial region; a feature modelapplication module configured to program the processor to apply afeature model, comprising both shape and texture components, to theextracted feature to obtain a first detailed match to said feature; afacial model adjustment module configured to program the processor toadjust the facial model, responsive to the first detailed match of thesecond model to said feature, to obtain an improved match of the facialmodel to the facial region; a feature model adjustment module configuredto program the processor to adjust the feature model, responsive to theimproved match of the facial model to said facial region, to obtain animproved match to the feature; a classification module configured toprogram the processor to extract principal shape components from boththe facial model and the feature model; and a facial expression moduleconfigured to program the processor to characterize said principal shapecomponents to determine a best matching facial expression of a discreteset of facial expressions.
 32. The device of claim 31, wherein thefeature comprises one or two eyes, an eye feature, one or two lips orpartial lips, or one or more other mouth features, or combinationsthereof.
 33. The device of claim 31, wherein the feature comprises ageometric feature.
 34. The device of claim 33, wherein the geometricfeature comprises a shape, deformation or location of one or more facialcomponents, or a pose variation, or combinations thereof.
 35. The deviceof claim 31, wherein the feature comprises an appearance feature. 36.The device of claim 35, wherein the appearance feature comprises a skintexture change.
 37. The device of claim 36, wherein the skin texturechange comprises a furrow, a bulge, an expression wrinkle, or anillumination variation, or blushing, or combinations thereof.
 38. Thedevice of claim 31, wherein the best matching facial expressioncomprises an indication of surprise, fear, happiness, anger, neutral,sadness, or disgust.
 39. The device of claim 31, further comprising agame adaption module configured to program the processor to adapt a gamedifficulty based on the best matching facial expression.
 40. The deviceof claim 39, wherein the game adaptation module is further programmed toadapt a game workflow based on the best matching facial expression. 41.The device of claim 31, further comprising a game adaption moduleconfigured to program the processor to adapt a game workflow based onthe best matching facial expression.
 42. The device of claim 31, furthercomprising a game adaption module configured to program the processor tomirror the best matching facial expression in an avatar within a gamedisplay.
 43. The device of claim 31, wherein the facial model comprisesan Active Appearance Model (AAM).
 44. The device of claim 31, whereinthe principal shape components comprise seven or less principal shapecomponents.
 45. The device of claim 31, wherein the principal shapecomponents comprise six or seven principal shape components.